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FINAL  TECHNICAL  REPORT 


GRANT  #:  N00014-01-1-0917 

PRINCIPAL  INVESTIGATORS:  James  Shanteau  (e-mail:  Shanteau@ksu. edul  Clive  J. 
Fullagar  (e-mail:  Fullagar@ksu. eduL  and  Scott  Hemenover  (email:  Shemenov@ksu.edu). 

IN  STITUTION  S :  Kansas  State  University 

GRANT  TITLE:  Selecting  and  Classifying  the  “Good  Sailor”:  Exploring  the  Non- 
Cognitive  Predictors  of  Expert  Team  Performance  in  Complex  Technological  Contexts 

AWARD  PERIOD:  1st  October,  2001  -  30th  September,  2002 

OBJECTIVE:  The  research  will  identify  non-cognitive  factors  that  are  associated  with 
competencies  that  are  predictive  of  expert  team  performance.  We  will  develop 
descriptive  quantitative  models  based  on  these  factors.  This  will  have  significant 
implications  for  the  selection  and  classification  of  Navy  personnel  who  will  inevitably  be 
working  in  teams  on  technologically  complex  tasks. 

APPROACH:  The  research  used  the  Controller  Teamwork  Evaluation  and  Assessment 
Methodology  (C-TEAM  computer  microworld)  that  was  developed  by  the  Human 
Resources  Research  Division  of  the  Civil  Aeromedical  Institute  at  the  FAA.  C-TEAM  is 
a  multisector  research  platform  designed  to  simulate  radar-based  air  traffic  control  tasks. 
The  C-TEAM  research  platform  consists  of  five  (four  clients  and  a  remote  server) 
80486/DX2  personal  computers.  C-TEAM  includes  a  scenario  generator  that  allows  the 
experimenter  to  create  scenarios  and  incorporate  a  large  number  of  experimental 
manipulations.  Several  measures  of  performance  are  automatically  tracked  by  the 
software,  including  separation  errors,  standard  protocol  deviations,  and  percent  of  aircraft 
reaching  destination.  This  performance  data  is  automatically  written  to  a  replay  file.  The 
replay  data  file  is  detailed  enough  to  allow  playback  of  the  performance  in  its  entirety, 
including  all  control  actions  issued  by  the  sector  controllers.  The  software  allows  sector 
controllers  to  communicate  with  each  other.  The  controllers  can  coordinate  handoffs, 
issue  restrictions,  and  deliver  other  inter-sector  communications  using  computer- 
mediated  communication. 

Our  approach  derives  from  previous  theory  and  research  suggesting  that  certain  team 
competencies  predict  effective  team  performance.  Our  model  extends  that  of  previous 
work  on  expert  teams  by  (a)  proposing  that  non-cognitive  individual  difference  variables 
will  predict  the  development  of  team  competencies,  and  (b)  testing  these  relationships  in 
a  complex  technological  environment.  Several  novice  and  expert  teams  will  perform 
simulations  of  varying  difficulty  levels  using  C-TEAM,  a  computer  microworld 
simulation  of  an  air-traffic  control  environment.  We  will  examine  the  relationships 
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among  individual  difference  variables,  and  the  development  of  team  competencies  and 
team  performance  over  time. 

CONCLUSIONS:  There  are  several  conclusions  that  can  be  drawn  from  the  research  to 
date: 

(1)  The  use  of  computer  microworld  environments  (such  as  C-TEAM)  facilitates 
research  on  teams  by  enabling  longitudinal  study  of  team  performance  in  complex 
technological  environments. 

(2)  Personality,  specifically  conscientiousness,  does  predict  team  performance. 
However,  a  “team’s  personality”  must  be  represented  in  such  a  way  as  to  reflect 
the  nature  of  the  task  performance.  By  considering  the  nature  of  the  task 
performance,  and  aggregating  predictors  accordingly,  personality  was  found  to  be 
a  strong  predictor  of  team  performance. 

(3)  Results  from  the  research  offer  consistent  and  strong  support  for  the  causal 
inference  that  team  performance  predicts  team  cohesiveness  and  not  vice  versa. 
This  contradicts  much  theory  and  research  that  has  suggested  that  performance 
precedes  cohesiveness.  We  have  discovered  that  teams  do  not  have  to  be  cohesive 
to  start  with  in  order  to  perform  well.  Providing  teams  with  tasks  that  they  can 
succeed  in  is  a  more  effective  way  to  increase  cohesiveness. 

(4)  The  research  also  indicates  that  more  effective  teams  tend  to  develop  more 
cohesive  or  similar  mental  models,  specifically  in  terms  of  the  way  in  which  team 
members  approach  the  task.  We  see  this  as  an  important  area  for  future  research. 

(5)  The  research  has  developed  a  measure  of  expert  team  performance  based  on  the 
Cochran- Weiss-Shanteau  (CWS)  index  of  individual  expertise.  CWS  assumes  that 
expert  teams  can  discriminate  among  similar  (but  not  equivalent)  stimuli,  and  that 
they  are  consistent  in  their  judgments  of  identical  stimuli.  The  team  CWS 
measure  was  successful  in  capturing  changes  in  team  performance  on  tasks  of 
varying  complexity.  It  was  also  correlated  with  other  measures  of  team 
performance,  and  capable  of  tracking  team  performance  longitudinally. 

(6)  Many  team  tasks  involve  maintenance  activities  that  can  be  monotonous  in  nature. 
The  research  also  investigated  individual’s  proneness  to  boredom  and  it’s 
prediction  of  actual  boredom  on  the  task.  The  relationship  between  boredom  and 
the  number  of  errors  was  also  measured  for  both  individuals  and  teams.  Data  are 
currently  being  analyzed. 

SIGNIFICANCE:  The  current  research  extended  previous  research  by  going 
beyond  predicting  initial  performance  to  study  the  forecasting  of  expert  team 
performance  in  a  complex  decision-making  and  problem-solving  environment.  Currently 
the  primary  instrument  for  selection  and  classification  of  Navy  personnel  is  the  ASVAB. 
The  ASVAB  test  battery  consists  often  subtests  representing  four  cognitive  abilities: 
Verbal  Ability,  Mathematical  Ability,  Technical  Knowledge,  and  Perceptual  Speed. 

While  there  is  considerable  evidence  of  the  predictive  validity  of  such  cognitive  factors, 
there  is  a  need  to  assess  whether  non-cognitive  variables  can  explain  additional  variance 
in  task  performance.  The  current  research  studied  the  relationships  between  several  non- 
cognitive  factors  and  team  competencies  and  performance.  The  goal  was  to  determine  the 


extent  to  which  these  non-cognitive  variables  could  explain  individual  and  team  task 
performance  beyond  cognitive  factors. 

There  has  been  a  shift  in  the  content  and  scope  of  jobs  defined  by  current 
classification  models  ( Sailor  21,  p.28).  Current  selection  and  classification  procedures  are 
sufficient  for  selecting  individuals  based  on  isolated  component  abilities.  In  the  future, 
however,  sailors  will  need  to  be  able  to  effectively  perform  in  teams  on  dynamic  and 
complex  tasks.  Consequently,  selection  and  classification  tools  need  to  be  developed  to 
predict  those  competencies  necessary  to  perform  these  team  tasks. 

As  noted  above.  Navy  force  structure  is  in  the  midst  of  profound  structural 
changes.  Tasks  are  becoming  technologically  more  complex  and  are  increasingly 
performed  in  teams.  One  implication  of  these  changes  is  that  selection  and  classification 
procedures  will  have  to  be  redefined  to  identify  knowledge,  aptitudes  and  skills  that  are 
required  to  operate  in  these  new  task  environments.  The  current  study  assessed  several  of 
those  competencies  that  are  associated  with  individual  and  team  task  performance  in 
complex,  dynamic  technological  contexts. 

In  sum  then,  the  benefits  of  the  proposed  research  to  the  Navy  are: 

•  Improve  recruitment  procedures  by  identifying  those  knowledge,  skills, 
and  attitudes  that  are  associated  with  effective  team  performance. 

•  Reduce  the  costs  associated  with  early  attrition  by  selecting  recruits  with 
the  necessary  skills  and  personality  characteristics  to  match  the  kinds  of 
technologically  complex  and  team  environments  that  they  will  be 
operating  in. 

•  Improve  the  quality  of  new  recruits  by  going  beyond  looking  at  selection 
and  classification  procedures  based  solely  on  cognitive  factors. 

•  Improve  job  satisfaction  by  ensuring  that  sailors  are  selected  and  classified 
into  jobs  with  the  appropriate  skills,  attitudes  and  competencies. 

•  Improve  future  fleet  readiness  by  selecting  on  the  basis  of  skills  and 
attributes  more  suited  to  the  upcoming  technological  and  team  climate. 

One  of  the  products  of  the  current  research  is  the  development  of  a  microworld 
protocol  that  enables  research  on  teams  performing  in  technologically  complex 
environments.  This  has  enormous  implications  for  selection  and  training  in  both  the 
military  and  industrial  worlds.  As  technology  becomes  more  sophisticated  and  more 
reliance  is  place  on  teamwork,  research  methodologies  such  as  that  being  developed  at 
Kansas  State  University  will  become  invaluable.  Our  long-term  goal  is  to  use  these 
research  insights  to  develop  several  tools  that  can  be  used  to  identify  individuals  with  the 
best  potential  to  perform  well  in  a  team  environment. 


AWARD  INFORMATION:  The  grant  has  funded  four  graduate  students  and  three 
undergraduate  students.  The  research  has  generated  one  Masters  Thesis  and  one  Doctoral 
Dissertation. 
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